Ruling out a fourth generation using limits on hadron collider Higgs signals 
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We consider the impact of a 4th generation on Higgs to 77 and WW, ZZ signals and demonstrate 
that the Tevatron and LHC have essentially eliminated the possibility of a 4th generation if the 
Higgs is SM-like and has mass below 200 GeV. We also show that the absence of enhanced Higgs 
signals in current data sets in the 77 and WW, ZZ final states can strongly constrain the possibility 
of a 4th generation in two-Higgs-doublet models of type H, including the MSSM. 
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Although new physics has not yet been seen at the 
Tevatron or LHC, as the integrated luminosity, L, esca- 
lates increasingly interesting constraints on new physics 
emerge. This Letter focuses on the interconnection be- 
tween limits on excesses in the 77 and WW, ZZ mass 
spectra and the possible existence of a 4th generation 
and/or a sequential W' , assuming existence of: (1) a 
Standard Model (SM) Higgs boson; or (2) a two-doublet 
Higgs sector (including the special case of the Minimal 
Supersymmetric Standard Model, MSSM) . Important re- 
sults arise even though a Higgs boson has not yet been 
detected. 

There are now significant constraints on Higgs to 77 
and WW signals coming from the current Tevatron and 
LHC data samples. A convenient review is Ref. py. In 
particular, no peak is observable in the 77 channel in 
the L = 131 pb~^ ATLAS data, and, indeed, the ob- 
served rate lies somewhat below the expected background 
rate. Similarly, both the LHC and, especially, the Teva- 
tron restrict any excess in the WW channel relative to 
the SM rate. We define the ratio i?^ = [T'/^gBR{h 

X)]/\r'^gl"BR{hsM X)], where the denominator is al- 
ways computed for 3 generations. Crude estimates from 
the ATLAS 77 spectrum plots of |T] are that R^-y < 10 for 
in the 100— 150 GeV range. As regards Rww, cur- 
rently the Tevatron CDF-t-DO combination ^ provides 
the strongest limits: at 95% CL the Bayesian upper lim- 
its on Rww in the nih & [100, 200] GeV window range 
between 2.54 and 0.64. Limits of this same order will 
eventually be achieved out to large mww L increases. 

These constraints motivate an examination of the pos- 
sibilities for enhanced i?^^ and Rww values in the con- 
text of various models for the Higgs sector. Here, we con- 
sider implications for a 4th generation in the context of 
the Standard Model (SM) and two-Higgs-doublet models 
(2HDM) (including the MSSM) and for a sequential W' 
in the SM case. The lepton and quark masses of the 4th 
generation will be set to 400 GeV and 1400 GeV will be 
chosen for the W' mass, both only slightly above current 
experimental limits. 

A plot showing R^~f and Rww as a function of rrih 
in the case of an h with SM-like couplings and decays 
appears in Fig. [I] (See also [3].) If a 4th genera- 
tion is present, one observes large R^^ (> 4) only for 
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FIG. 1: The solid black curve shows Rww in the presence 
of a 4th generation. For i?^^: the long-dash - short-dash red 
curve is for a 4th generation only; the dotted blue curve is for 
a sequential W' only; the long-dash magenta curve is for a 4th 
generation plus a sequential W' . All curves are for a Higgs 
boson with SM-like couplings and SM final decay states. 

rrih > 2mw,^ where, in any case, prospects for probing 
Rjj < 4 must be regarded as uncertain due to the large 
size of the Higgs total width. Fortunately, the WW chan- 
nel is much more definitive. Rww, ^ilso plotted in Fig. [T] 
is predicted to be > 6.5 for rrih < 300 GeV, falling to 
> 4.8 for mh € [400, 500] GeV. This is in clear contra- 
diction to the above quoted experimental limits from the 
Tevatron for the [110,200] GeV mass range. Thus, the 
WW channel already implies that having a light SM- 
like Higgs boson is inconsistent with the presence of a 
4th generation. (See also the earlier analysis of [1] using 
less integrated luminosity.) The only escape would be 
if the Higgs boson has non-standard decays that deplete 
BR{h WW) and BR{h 77). Since models of this 
type abound [5 , a definitive conclusion will require ac- 
tual observation of a Higgs with the couplings and decays 
predicted in the SM. 

Before leaving the SM, we note from Fig. [T] that in- 
clusion of a heavy sequential W' without a 4th genera- 



Rj-y ~ 1 for rrih fa 130 GeV because the increase in Tgg is closely 
offset by a decrease in BR{'yy) resulting from the increased can- 
cellation of the 4th generation fermion loops with the (opposite 
sign) W loop. 
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TABLE I: Summary of 2HDM quark couplings in Model I and 
Model II. 
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tion gives R^^ ~ 4 — 5 for rrihsM ~ 115 GeV, a value 
that can probably be excluded relatively soon. But, once 
TO/tsAf ~ 2tovv ^77 falls to '-^ 3, a value requiring large L 
to either observe or exclude given that rf^^t" is large for 
such masses. If both a 4th generation and a sequential 
W are present the predicted R^^ ~ 15 — 20 is probably 
already excluded for rnhg^j ^ 150 GeV (perhaps higher 
once the analysis is done) using the current data set. In 
contrast, Rww is nearly unaffected by a possible W. 

Even more enhanced signals from the Higgs bosons of 
the 2HDM are very possible. In the context of the 2HDM 
(a convenient summary appears in the HHG the 
masses of the light and heavy CP-even Higgs bosons, h 
and H, of the CP-odd Higgs boson, A, and of the charged 
Higgs boson as well as the value of tan f3 (the ratio of 
VEVs for the two doublets) and the CP-even Higgs sector 
mixing angle a can all be taken as independent param- 
eters, whose values will determine the of the general 
2HDM Higgs potential. Thus, it is appropriate to present 
results for each neutral Higgs boson as a function of its 
mass for various tan/3 values. 

As reviewed in [6J, in the 2HDM there are only two 
possible models for the fermion couplings that naturally 
avoid flavor-changing neutral currents (FCNC), Model I 
and Model II. As a brief reminder, we provide the sum- 
mary of Table[T]of the couplings of the h, H and A in the 
two cases, relative to SM normalization. In both Model 
I and Model II the WW, ZZ couplings of the h and H 
are given by sin(/3 — a) and cos(/3 — a), respectively, rel- 
ative to the SM values. And, very importantly, there is 
no coupling of the A to WW, ZZ at tree level. If the A; 
of the Higgs potential are kept fully perturbative, the de- 
coupling limit, in which ttih — > rriA and sin^(/? — a) — > 1, 
sets in fairly quickly as ttia increases 

In this Letter, we focus on the 2HDM-II coupling pos- 
sibility, and the CP-odd A, for which only 77 decays are 
relevant. R^^ is plotted as a function of uia in Fig. [2] for 
the 3 generation case. Enhanced 77 signals, R:^^ > 1, are 
only possible for low tan /3 values. Although not shown, 
enhanced signals are possible for tan (3 < I also in Model 
I. Note that R^^ is not influenced by possible sequential 
W's since they do not couple to the A. 

The impact of a fourth generation on the two-doublet 
results depends strongly on whether or not the model is 
Model I or Model II. In particular, a 4th generation does 
not affect R^^ in the case of Model-I. This is because 
the t' and b' of the 4th generation couple to the A with 
opposite signs but equal coefhcients — see Table [T] In 
contrast, the results for a Model-II A are changed dra- 
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FIG. 2: R-.,-., for the 2HDM-II A. The legend is as follows: 
solid black— >■ tan/3 — 1; red dots— >■ tan/3 = 1.5; solid red— > 
tan/3 = 1/1.5; cyan dots— >■ tan/3 — 2; solid cyan— > tan/3 = 
1/2; green dots— >■ tan/3 = 3; solid green— > tan /J = 1/3; ma- 
genta dots— >■ tan/3 = 1/5; solid magenta— > tan/3 — 5;blue 
dots— >■ tan/3 — 10; solid blue— > tan/3 — 1/10; long red dashes 
plus dots— > tan/3 — 30; pure long red dashes— >■ tan/3 = 1/30; 
black dotdash— >■ tan /J — 50. This and subsequent figures 
must be viewed in color in order to resolve the different tan /3 
cases. 
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FIG. 3: R-y^ for the 2HDM-II A after inclusion of 4th gen- 
eration loops in gg production and in y4 — )■ 77 decays. The 
legend is as in Fig. [2] 

matically: the 4th family case is illustrated in Fig. [3] 
Regardless of tan /3, one predicts large R:^^, the smallest 
values occurring at low niA for moderate tan/3 G [1,5], 
for which R^^ - 10 for tua G [30, 150] GeV. Of course, 
this is precisely the range of tan/3 that is preferred in 
order that the Yukawa coupling of the t' is perturbative. 
R^^ increases dramatically for niA > 2m\Y because of the 
drop in BR{hsM — >■ 77)- The enhanced values of R^^ 
are least likely to be depleted by A decays to non-SM fi- 
nal states, most particularly A hZ, W^ , when tua 
is not large. 

As noted earlier, a rough estimate using the latest AT- 
LAS plot shown in [Tj suggests R^^ < 10 for M^^ < 
150 GeV. This estimate assumes a narrow resonance. 
A plot of F^t for niA < 500 GeV is given as Fig. ^ 
for the 4 generation case. Since the t' and b' masses 
are larger than itia/'^, direct decays to 4th generation 



3 



2HDM-II, 11^=4, n^=l 




200 300 
(GeV) 



500 



FIG. 4: Ftot for Model II after inclusion of 4th generation 
loops for A — ^ gg, 77 decays. The legend is as in Fig. [2] 
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FIG. 5: rrih vs. ttt-a for tan /3 = 1.5, 2 and 3 — legend as in 
Fig. [2] 



quarks do not occur, but the 4th generation quarks do 
influence the loop-induced decays to gg (and 77). For 
TTiA < 150 GeV, the narrow width approximation only 
breaks down for tan f3 > 30. At mA = 150 GeV, 
r^„t = 5 GeV, 13 GeV for tan/3 = 30,50, respectively 
For such total widths, limits would then be weaker than 
naively estimated using the narrow resonance assump- 
tion. However, we should note that tan/3 > 30 is ex- 
cluded by LHC data for < 170 GeV ^ using the 
A — )■ r+T" decay mode and just L = 35 pb"^ of data W]. 
These limits will improve very rapidly with increased L. 
Once TUA > 'ivrit the A total width increases dramati- 
cally; a study of the feasibility of detecting a highly en- 
hanced broad 77 signal above the continuum 77 back- 
ground is needed to determine the level of sensitivity. 

In passing, we note that R^^ and i?;^ for the CP- 
even Higgs bosons are less robust as indicators of a 4th 
generation — in particular, they depend significantly on 
sin^(/3 — a) and are often below 1 (especially for the 
Yukawa-perturbativity-preferred modest tan/3 values). 
However, it is important to note the complementary of 
^ww ^'^^ ^77 decoupling hmit of sin^(/3 — a) — 1. 

In this limit, it is R^^r that currently does and R^^^ that 
shortly could rule out a 4th generation scenario if the h is 
relatively light and if the A is not too heavy, respectively. 

Many possible scenarios at the LHC can be envisioned. 
For example, as L increases it could be that a light A 
{iriA < 200 GeV) is observed in the r+r" mode with 
rate corresponding to a modest tan /3 value (presumably 
below 30 given current limits). If there is no sign of a 
77 peak for the given L it could easily happen that the 
limit on R^^ will exclude a 4th family in the Model II 
context. If, on the other hand, no A is detected in the 
T+T^ mode a limit on tan/3 significantly below 30 in 
the niA < 200 GeV mass region is likely. In this case, we 
could only conclude that there can be no 4th generation if 
we assume the 2HDM Model II structure and that niA < 



^ This assumes the A and H are not degenerate. 



200 GeV; but, of course, no contradiction would arise if 
niA is significantly larger or if the 2HDM Model II is not 
the right model. 

Let us now focus on the MSSM. There are many stud- 
ies of the impact of a 4th generation on MSSM Higgs 
physics [5HTT]. Substantially larger masses than the 
400 GeV value we employ here are strongly disfavored 
by precision electroweak constraints and FCNC consider- 
ations [To]. For masses ^ 400 GeV, large 4th family loop 
corrections imply a large mass for the h while perturba- 
tivity for the 4th generation Yukawas requires tan /3 < 
2 — 3. As stated earlier, for given soft-SUSY-breaking 
parameters, all Higgs masses and branching ratios are 
fixed once uia and tan/3 are specified. For this study, we 
employed an extended version of HDECAY3.60 [T2] with 
"default" hdecay.in soft-SUSY-breaking inputs — 4th 
generation soft parameters are taken to be identical to 
those for the 3rd generation. The resulting values of to^ 
as a function of niA are plotted in Fig. [5j 

Once again, strong constraints on the possible pres- 
ence of the 4th generation arise from considering i?(VvF 
and R^^^. The relevant plots appear in Fig. |6] These 
plots include loop effects from both the fermions and the 
sfermions of the 4th generation, but the sfermion and 
other parameters of the default hdecay . in are such that 
all sparticles are heavy and do not contribute signifi- 
cantly to the h or A decays for iriA < 500 GeV. The 
smallest values of R^^ ^ 6.5 occur in the itia < 2rnn/ 
region. (Ref. [9] also finds enhancement in R^^ at low 
niA-) And, as for the 2HDM-II, for > 2m\Y one 
finds R:^^ > 100! Rww complementary in that for 
TTiA > 200 GeV, R'^yY > 2.4, a value that will be probed 
even at the large mww ~ tt^h values of Fig. [5] given large 
enough L at the LHC. 

Thus, we have the following situation. Analysis of LHC 
77 spectrum data will probably soon place a limit of 
R^^ < 6.5 out to TTiA — ~ 2miY, in which case a 
4th generation will be inconsistent with the MSSM for 
niA ^ 2mw, barring significant A — > SUSY decays. For 
2mw < niA < 200 GeV it seems likely that a limit be- 
low the minimum predicted value of R^^ = 100 will be 
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FIG. 6: MSSM plots for tan^ = 1.5, 2 and 3 — legend as in 
Fig. [2] Top: Rww vs. mA- Bottom: 7?^^ vs. niA- 



achieved. Meanwhile, for 4 generations Rl^y^ > 2.4 is 
predicted for all niA > 200 GeV and will eventually be 
excludable in the relevant m/j ~ 400 — 500 GeV mass 
range. If sparticles are light, then hopefully the LHC 
will detect them and R'^^r and R^^ predictions can be 
corrected for substantial BR{h, A — SUSY) values. In 
addition, predictions for T'^fBR{h, A SUSY) will be 
larger in the presence of a 4th generation than without. 



Finally, we note that if there is a W, R^^ is not af- 
fected (because of the absence of a tree-level AW'W 
coupling) while changes to R%/t^ are very tiny. Further, 
Rww only modestly influenced by sfermion loop con- 
tributions to and sfermion loops are not present for 
either gg — > A or A 77. Thus, R"^^ and ^''^ quite 

robust tests for the presence of a 4th generation and can 
potentially eliminate the possibility of 4 generations in 
the context of the MSSM even if no Higgs is observed. 
Of course, by the time sufficient L is available to measure 
Rww '-'^^ ^'^ large rrih, direct observation or exclusion of 
the 4th-generation quarks may have occurred. 

Once a 77 or WW peak emerges (as will eventually 
happen if there is one ore more light Higgs bosons) a 
multitude of possibilities will need to be analyzed. If no 
Higgs has been seen in any other mode, then there will be 
a plethora of Higgs sector choices that could explain the 
77 or WW peak, both in the general 2HDM context and 
in the MSSM. In the MSSM context, if tan P is known 
from general observations of superpartners, it will be im- 
portant to see if there is a Higgs boson within some Higgs 
scenario that can explain the peak for the known tan /3 
value, either with or without a 4th generation and/or W' . 

To summarize, we have shown that great importance 
attaches to the most exhaustive possible search for peaks 
and enhancements in the 77, WW and ZZ mass spec- 
tra over the broadest possible range of M^^, raww-^ and 
mzz- Either detection of a peak or a simple limit on 
R^~f, Rww and Rzz as a function of M-^^, Mww, and 
Mzz will provide highly significant constraints and/or 
consistency checks both on the Higgs sector and on the 
possible existence of a 4th generation or W' . 
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